CONSTRUCTING STRONG MARKOV PROCESSES 



ROBERT J. VANDERBEI 
Dedicated to the memory of Lynda Singshinsuk. 

Abstract. The construction presented in this paper can be briefly described as follows: 
starting from any "finite-dimensional" Markov transition function p(, on a measurable 
state space {E,B), we construct a strong Markov process on a certain "intrinsic" state 
space that is, in fact, a closed subset of a finite dimensional Euclidean space M . Of course 
^! we must explain the meaning of finite-dimensionality and intrinsity. Starting with pt, we 

^ consider the range of the nonnegative bounded measurable functions under the action of 

the resolvent. This class of functions induces a uniform structure on E. We say that E 

is finite-dimensional if this uniformity is finitely generated. In such cases we then map 

p<^ E into M'^. The intrinsic state space is the closure of the range of this mapping. On 

this enlarged state space we construct a strong Markov process, which corresponds quite 
naturally to pt- We give several examples including the usual examples of nonstrong 
Markov process. 



1. Introduction and notation 

O 

Every student of probability learns very early that not every Markov process possesses 
^ the strong Markov property. And, as is often the case, the simplest pathological examples 

fT^ are very simple indeed. For instance, the process that moves deterministically to the left 

^ on the real line with unit speed except that when it reaches zero it pauses there for an 

^ exponential amount of time before continuing, is perhaps the best such example. The 

^ ^ fact that it is a Markov process follows from the mcmoryless property of the exponential 

^ random variable. To see that it is not a strong Markov process, consider the first hitting 

time of the open left half-line. Starting from this time, the process proceeds immediately 
into the left half-line. However, at this random time the process is situated at the origin 
and consequently if it had the strong Markov property it would have to remain there for 
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an exponential amount of time. The fact that it does not shows that it is not strong 
Markov. 

Of course the above pathology can be remedied by splitting into two points 0+ and 
0^. If we put the exponential alarm clock at 0"*" and have the process immediately appear 
at 0^ when the clock rings (i.e. a right continuous trajectory) then in this enlarged state 
space the process is strong Markov. 

This idea of enlarging the state space first appeared in the paper [15] by Yushkevich. 
Shortly after that, D. Ray published a fundamental paper [H] in which he introduced a 
certain compactification of the state space on which the Markov process becomes what 
is today called a Ray process (which has the strong Markov property). Ray's methods 
were clarified and improved by F. Knight [I3]. The short monograph [H] by Getoor has 
become the standard reference on Ray processes and the Ray-Knight compactification. 
However, the most general results appeared later in the papers [SISIEIEIEIIHIIS] ofH. 
J. Engelbert. His basic assumption is that there exists a certain class of nonnegative 
bounded measurable functions, on a countably generated measurable state space, that 
forms a separable Ray cone (see Definition 12 in [S]). 

The starting point of this paper is a Markov transition function on a measurable state 
space. We introduce in a canonical way a certain class R+ of functions that satisfy most 
of the requisite properties of a Ray cone. The class R+ is generally not separable (even 
for the above example) but is usually "finite dimensional." This means that there exists 
a finite collection E = {ei, 62, ... , e^} of real- valued functions (not necessarily bounded) 
such that, for every sequence of points x„ in the state space E, f{xn) is Cauchy for every 
/ G R+ if and only if ej (x„) is Cauchy for i = 1, 2, . . . , d. By thinking of the functions Cj 
as coordinate functions, the map x — )■ (ei(a;), 62(0;), . . . .Cd^x)) embeds the state space in 
W^. We then take the closure of the range of this mapping to get our intrinsic state space. 
Thinking in terms of our original state space E, this closure corresponds to a completion 
of the state space relative to the uniformity generated by R+ (for a general discussion 
of uniformization see e.g. |T2] Chapter VI). On the resulting closed space we construct 
a strong Markov process that corresponds naturally to the original Markov transition 
function. 

The advantage of our approach is that the construction is canonical. The Ray-Knight 
compactification, on the other hand, depends on the separable Ray cone with which one 
starts. In fact a substantial part of the theory of the Ray-Knight compactification consists 
in studying to what extent the new state space depends on the Ray cone (see e.g. [TT] , 
Chapter 15). 
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The main goal of this paper is the construction of a process with the strong Markov 
property. Along the way we get some other results; e.g. right continuity and the existence 
of left limits (recently this has been called cadlaguity — a gauche term indeed). Since the 
state space we construct is a closed subset of M.'^, the further sample path properties are 
for the most part immediate consequences of already well known results and so they will 
not be pursued here. 

The remainder of this section is devoted to establishing notations that will be used 
throughout the rest of the paper. 



Definition. A kernel k on a. measurable space {E, B) is a real- valued function of E x B 
satisfying: 

1.1 X — )■ k{x,B) is i3-measurable; 

1.2 B — > k{x, B) is a signed measure having finite total variation measure \k\{x, ■). 



The formula 

1.3 \\k\\ = sn^\k\{x,E) 

X 

defines a norm on the space K of all kernels for which \ 1.3 ) is finite. It is easy to check 
that i^T is a Banach space. The formula 

k*k'{x,B) = j k{x,dy)k'{y,B) 
makes K into an algebra. We denote by A;*^"-* the n*'^ power of the kernel k. We also put 



k^^\x,B) = l{x,B) 



1 X e B 
X ^B. 

It is easy to see that ||A; * A;'|| < || || A;'|| and hence that < HA;!!". A kernel k is 

called a stochastic kernel if, for every x E E, k{x, ■) is a probability measure. 

If {E, B) is a measurable space, we also denote by B the Banach space of bounded 
i3-measurable functions on E and we denote by B^ those f E B that are nonnegative. 
The norm on B is given by ||/|| = sup^g^; 1/(^)1- For every kernel (say k) on {E,B), we 
denote by the corresponding upper case letter {K in this case) the linear operator on B 
defined by 

Kf{x)= [ k{x,dy)f{y). 



If 5 — )■ k{x, B) is a positive measure then is a positive operator and if /c is a stochastic 
kernel then Kl = 1 and \\Kf\\ < \\f\\. 
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If is a subset of M'^, we denote by C^{E) the collection of all bounded continuous 
functions on E that have limits on the closure of E and that are k times continuously 
differentiable with each derivative being bounded. We will write C{E) instead of C^{E). 
The notation / G C^{Ei, E2, . . . , En) indicates that, for each j, the function / restricted 
to Ej is of class C^{Ej). The notation Cq{E) denotes those functions in C{E) that tend 
to zero at infinity. Hence if E is bounded, Co{E) = C{E). Finally E denotes the closure 
of E. 



2. The intrinsic state space 

Definition. A family of kernels {pt)t>o on a measurable space {E, B) is called a Markov 
transition function if: 

2.1 (t, x) -^pt{x,B) is i3(]R'^) X B measurable; 

2.2 every pt is a stochastic kernel; 

2.3 ps+t = Ps* Pt- 



The space {E, B) is called the state space for pt. 



Definition. A family of kernels ('U°)q.>o on a measurable space {E, B) is called a Markov 
resolvent if 

2.4 for every a > 0, -^u"" is a stochastic kernel; 

2.5 - = (a - * m^. 



If Pt is a Markov transition function, the formula 

POO 

= / e-^^ptdt 



defines a Markov resolvent. 

For the rest of this section, let pt denote a fixed Markov transition function on a fixed 
state space {E, B) and let be the corresponding resolvent. We put 

C = f/"i3. 

To see that the right hand side does not depend on a (as long as a > 0) rewrite property 
( 2.5) as = u"' * [1 + {a — (3)u^]. Hence, if / = U^g and g & B, then / = U"h where 
h = g + {a — (5)U^g clearly also belongs to B. Put 

C+ = U f/"i3+. 

a>0 
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The following properties of C and are easily verified: 

2.6 f/"C+ C C+, f/"C C C; 

2.7 is a convex cone, C is a linear space; 

2.8 contains the nonnegative constants; 

2.9 C = C+ -C+. 

Definition. A function / is called a-supermedian if / G and / > (/3 — a)U^f for all 
/3 > 0. 

2.10 Proposition. The following properties hold: 

(i) every / G C+ is a-supermedian for some a > 0; 

(ii) for every / G C, lim^o \\Ptf - f\\ = 0. 

Proof. Property (i) is an immediate consequence of the definition of and the resolvent 
property ( 2.5[ ). To prove (ii), put g = U°'f. It is not hard to show that 



-at i 



Hence 







0<\\e-'''Ptg-g\\ <t 
and so 

< \\Ptg-g\\ 

< WPtg-e-'^'PtgW + We-^'Ptg-gW 

< il-e-'^')\\g\\+t\\f\\. 

Letting t tend to zero establishes (ii). □ 

It is easy to see that the minimum of two a-supermedian functions is again a-supermedian. 
Also convex combinations of a-supermedian functions are again a-supermedian. 

2.11 Theorem. There exists a unique minimal convex cone such that: 

(i) C+ C 7^+ C B+- 

(ii) for a > 0, f/"7^+ C 7^+; 

(iii) f,gE 71+ implies that f A g E 71+ . 

In addition, every / G 7Z+ is a-supermedian for some a > 0. 
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Proof. If 1-L is any convex cone in B+, we define the usual enlargements U{l-L) and A('H) 
as follows: 



It is easy to check that U{l-L) and A('H) are again convex cones in . Also the sum of 
two convex cones is again a convex cone. 

Let TZq — C"*" and define TZn recursively by TZn — A(7?.„_i + U {TZn-i))- Put TZ^ — U TZn- 

n 

Note that TZn C TZn+i for each n > 0. By induction on n, it follows that each Tin is a 
convex cone in and that every / G TZn is a-supermedian for some a > 0. Hence 7^+ 
also has these properties. If f,g E TZ+ then there exists an n > with f,g & TZn- Then 
f A g E TZn+i C TZ+. Also, if / G TZn then f/"/ G 7^„+i C 7^+. It is obvious from the 
construction that TZ+ is the unique minimal convex cone that satisfies (i)-(iii). □ 

2.12 Definition. Two classes TL and Q of real- valued functions defined on a set E are 
said to generate the same uniformity on E if, for every sequence Xn G E, the following 
are equivalent: 

(i) h{xn) is a Cauchy sequence for every h eTL, 

(ii) g{xn) is a Cauchy sequence for every g E Q. 

We write Ti ^ Q to indicate that Ti and Q generate the same uniformity. We say that 
the uniformity generated by a class Q is finitely generated if there exists a finite collection 
TL such that TL ^ Q. We will only consider finitely generated uniformities (which is why 
wc do not need to introduce nets). 

The basic example to keep in mind is the following: let E = M"^, TL be the set of 
coordinate functions and Q be the set of bounded continuous functions. Then Ti ^ Q and 
since Ti is finite, Q is finitely generated. In this case h{xn) is Cauchy for all /i G ?^ if and 
only if Cauchy sequence relative to the Euclidean metric. 

It is easy to sec that the relation ~ is transitive: Ti ^ Ti' and Ti' ~ Ti" implies that 
Tir^Ti\ We also have 

2.13 Proposition. UTL ^ Q, then Ti separates points x and y if and only if Q does. 



U{Ti) 

m) 



W'fi + ■■■ + f/""/n I > 0, /, eH,l<3<n,n>l} 
{fiAf2A---Afn\fj^n,l<j<n, n>l}. 
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Proof. If h{x) = h{y) for all /i G H then {h{x), h{y), h{x), h{y), . . .) is a Cauchy sequence 
for every h E Ti. Hence {g{x) , g{y) , g{x) , g{y) , . . .) is a Cauchy sequence for every g E Q 
and so g{x) = g{y) for all g E Q. □ 

Put 71 = 71+ — 7Z+. We are now ready to introduce our basic assumption on p^: 

2.14 Assumption. There exists a finite collection S = {ei, 62, . . . , Cd} of real-valued 
functions on E (not necessarily bounded) such that S ^ 7Z. 

The examples in Section 4 show that this condition is often readily verifiable. In analogy 
with the example given above, it is good to think of functions in TZ as bounded continuous 
functions and those in S as coordinate functions. With this in mind we see that I 2. 141 is 
the assumption that the state space is finite dimensional. 



In assumption 2.14, it is clear that we could replace 7Z by 7^+. Define a map ip from 
E into R'^ by 

ij{x) = (ei(x), e2(x), . . . , erf(x)) . 



The mapping ^p may fail to be injective (see Example 4.8). In any case, we will abuse 
language and refer to ip as an embedding of E into M'^. Even if ijj is not injective, we have 
the following result. 

2.15 Proposition. If 'ijj{x) = ilj{y) then f{x) = f{y) for all f eTZ. 



Proof. Immediate consequence of Proposition 2.13 □ 



It follows that is injective if and only if TZ separates points. 
2.16 Proposition. Suppose that £ = {ei,e2, . . . , e^} and £' = {e[, 63, . . . , e^,} satisfy | 



2.14 and let ip and ip' be the corresponding embeddings. Then there exists a continuous 



bijection between "tpiS) C M'^ and ip'{£) C M.^' whose inverse is also continuous. 



Proof. Since ~ is transitive we see that £ ~ £'. Hence, it follows from proposition 2.13 
that ip'oip"'^ and ipo^ip')'^ are well defined. It is also clear that Xn is a Cauchy sequence in 
il'{£) if and only if = ip' oip'^^Xn) is Cauchy in ip'{£). Hence ip'oip'^ and ipo^ip')'^ can 
both be extended to the closures of their domains and give us the desired bijections. □ 
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Proposition | 2.16| shows that our construction is canonical and that the specific choice 



of £ is just a matter of choosing coordinate functions. 



Proposition 2.15 shows that the following formula defines unambiguously a mapping 



from TZ into the space of real- valued functions on ^^{E): 

(M//)(0 = /(^-'(0), ee^(^), /G7^. 

The mapping \l/ is linear and injective. Also note that \l/ej makes sense and is the i^^ 
coordinate function on ip{£). 

2.17 Proposition. For every f e TZ, f is a continuous function on ip{S). 

Proof. Suppose that — ^ C,- Choose Xm, and x in E such that Xm G 'ip'^i^m) and 
X G 'ip^^iO- Then (ej(xi), ej(x), ei(x2), ei(x), . . .) is a Cauchy sequence for each i. By 



assumption | 2.14 {f{xi),f{x),f{x2),f{x),...) is a Cauchy sequence for every f E 71. 



Hence f{xm) — ^ f{x) for every f eTZ and so f{^rn) — ^ ^/(O every f eTZ. □ 



For each / G 7^ we can extend the definition \1'/ to ip{S) by continuity. To see this, 
suppose that — ^ ^ and ^ Then ej(xm) is Cauchy for each i = l,...,n 



where Xm is chosen from ip ^(^m)- Assumption 2.14 then implies that f{xm) is Cauchy 



for every f E TZ. Denote this new function by Put TZ' = : / G TZ}, 

IZ'j^ = {^/ : / G 7^+} and E' = "^{S). The space E' is called the intrinsic state space 
for pt. The class 7Z'_^_ is a convex subcone of C'^{E') that contains the nonnegative constant 
functions and is closed under pointwise minimization. Also TZ' = TZ'_^_ — TZ'j^ so it follows 
that TZ' is a linear subspace of C{E') which contains the constant functions and is closed 
under pointwise minimization and maximization. We will require one final property of TZ' 
which we state Lemma. 



2.18 Lemma. The class TZ' separates points in E'. 

Proof. Suppose that ^,ri E E' and ^/(O = ^^fiv) / ^ 7^- Let be a sequence 

of points in ip{S) that converges to ^ and let rjn be a sequence converging to rj. Then 
lim„ \l//(^„) = lim„ fijjn)- Let x„ be a point in ip~^{in) and let y„ be a point in ip'^irjn)- 
Then we have lim„ = lim /(?/„). Hence, the sequence (/(xi), /(xa), /(t/a), • • •) 

is Cauchy for every f E TZ. By assumption 2.14[ (ej(xi), ej(?/i), ej(x2), ej^/a), • • •) is 
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Cauchy for i = l,...,d. Hence lim„ej(a;„) = limn ei{yn) for all i. This means that 
lim^„ = lim?7„ and so C, = tj. □ 

For each a > 0, define a positive linear operator on TZ' as follows: 

: Wf ^^f f ^ ^(t/"/) ^ ^([/"/). 

The first map is simply the restriction map. The second map is \E'~^, which exists since 
\1/ is an injection. The fourth and fifth maps make sense since If^TZ C TZ. It is easy to 
see that V"" inherits the following properties from ?7": 

(i) aVn = 1; 

(ii) - 1/^ = (/3 - a)V"V'^; 

(iii) if J, pointwise and /„ G 71', then V^fn i pointwise. 

By Daniel's theorem (see e.g. [2] Section III-35), for each x G E', and a > 0, there exists 
a unique measure z/"(x) on (y{lZ') such that 

y"/(x) = j u"ix,dy)fiy), fen'. 

Since n' separates points in E' and E' is closed, it follows from Proposition | 5.1| that 
<y{n') = B{E'). From the lattice version of the monotone class theorem (see e.g. [2], 
Theorem 22.3) and the fact that x — )■ z/"(x, /) is B{E') measurable (even continuous) for 
every / e n', it follows that x — )■ z/°(x, /) is B{E') measurable for every bounded B{E') 
measurable /. Another monotone class argument shows that is a resolvent. 

In the next section, we start with a state space E, a family of real-valued functions 
n^ on E, and a resolvent which all together satisfy the conditions which we have 
found that E', n'_^ and z/" satisfy. From these we show that such a resolvent gives rise 
to a unique Markov transition function and that from this transition function we can 
construct a strong Markov process in the space of right continuous trajectories that have 
left hand limits. 

3. From resolvent to strong markov process 

In this section, we take as our starting point properties of the intrinsic state space that 
were developed in Section |2} Hence we make the following assumptions. Let {E,B) be a 
closed subset of M'^ with the Borel cx-algebra. Suppose that there is given a convex subcone 
7^+ of C~^{E) that contains the nonnegative constant functions and that separates points 
in E. Finally, suppose that we have a Markov resolvent f/" on {E, B) with the following 
properties: 
3.1 f/"7^+ C 7^+; 
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3.2 every / G 7?.+ is a-supermedian for some a > 0. 

Put IZ = IZ^ — 71+ . We need one additional assumption that is not a carry-over from 
Section 2 (or maybe it is — this is an open problem): 

3.3 Co{E) C cl(7^). 

The notation cl(7?.) means the closure of TZ in the sup norm. If is a bounded subset 
of R'^ then Co{E) = C{E) and, by the lattice version of the Stone- Weierstrass theorem, 
C{E) = cl(7^). Hence, in this case assumption ( 3.3) is automatic. The examples in 



Section 4 show that, even when E is unbounded, assumption ( 3.3) is easily verified. A 



resolvent that satisfies (3.1) and ( 3.2) will be called an Engelbert resolvent. 



The assumptions in force in this section are very similar to those of a Ray resolvent. 
Indeed for a Markov Ray resolvent f/" one assumes that E is a compact metric space, B 
is the Borel cr-algebra, and 

3.4 W'CiE) C C{Ey, 

3.5 the continuous a-supermedian functions separate points in E. 

We have the following connection between Ray and Engelbert resolvents. 



3.6 Theorem. If _E is a compact subset of 
[/" is an Engelbert resolvent. 



and is a Markov Ray resolvent, then 



Proof. We must find a family 71+ with the desired properties. We may take 71+ to be 
the family of all continuous functions that are a-supermedian for some a > 0. For 



any / G B^, U°'f is a-supermedian and, by ( 3.4), f/" maps continuous functions into 



continuous functions. Hence U°'7Z+ C 71+. Property ( 3.5) says that 7Z+ separates points. 
It is clear that 7Z+ is a convex subcone of C^{E) and so the proof is complete. □ 



With the aim of constructing a strong Markov process, our first goal is to invert the 
Laplace transform to recover pt from f/°. 

3.7 Theorem. Let be an Engelbert resolvent. Then there exists a unique Markov 
transition function pt such that the following hold: 

(i) For all a > 0, / G B: U'^f = J^^ e-'^'Ptfdt. 

(ii) For all / G 71+, there exists a > such that e^'^^Ptf < f. 

(iii) For all x G -E, / G TZ, the mapping t — )■ Ptf{x) is right continuous on [0, oo). 
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Proof. We will need several general properties of resolvents and a-supermedian functions. 
For convenience, we have summarized these properties in the appendix. Fix / G 7?.+ . 
Then there is an a > such that / is a-supermedian. According to 5.3 the lim^-_j.oo(/3 — 
a)U^ f{x) exists. We denote the limit by /(x). Fix x E E and put g{f3) = f{x) — (/3 — 
a)U^ f{x). By 5.3, g is nonnegative and by ( 5.2) 



d 

dp 



g{(5) = n\{U^T[l-{(5-a)U^]f. 



Since / is a-supermedian and is a positive operator, we see that g is completely 
monotone. Also ^'(0) = (7(0+) = f{x) + aU^f{x), which might be infinite {U^f{x) = 
limis^QU^ f{x) exists for every / G B^). By the Hausdorf-Bernstein-Widder Theorem 



(see e.g. [TO], p. 439) there exists a positive measure Xx,aif,-) on [0, 00) of total mass 
f{x) + aU^f{x) such that, for ^9 > 0, 



3.8 



/(a;) _ (/3 _ a)U^f{x) = / e-^*A.,„(/, dt). 

J 10,00) 



Note that Xx,a{f, ■) does not charge {0}: 



A.,a(/,{0}) = lim / e-^'XxM^dt) 



lim 

/3— >-oo 



fix) -W- a)U^f{x) 



Put, for / G 7^+, 

3.9 Pi"V(a;) = e-* /" e-"'^A,,,(/, rfs). 

Then for each x E E, the mapping t — e~°'^P^°'^ f{x) is decreasing, right continuous and 

e-°*P/'^V(^) t / e~"'XxALds) = f\x) 
J {0,00) 
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as 1 1 0, since Xx,a{f, ■) does not charge {0}. Moreover, for /3 > 0, 



/»oo re 

Jo Jo 



JO 



1 /"°° 

1 



/(x)-/(x) + (/3-a)f/'^/(x) 



/3 — a 

Hence Pj;"^ does not depend on a (by the uniqueness of the Laplace transform) and so 
we will now denote it simply by Pj. 

Consider /i, /2, . . . , /„ G 71+ and nonnegative real numbers ri, r2, . . . , Put / = 

i^ifi + 1- rnfn- Choose a sufficiently large so that fi, f2, . . . , fn are all a-supermedian. 

Then / is also a-supermedian. Since 

n 

/(a;) _ (/3 _ a)U^f{x) = ^ r, [/, - (/3 - a)U^ ^x)] , /3 > 0, 

it follows from the uniqueness of Laplace transforms that \x,a{f,-) = SiLi ^^i-^^^.^l/*' ')• 
It then follows from ( 3.9) that Ptf{x) = J27=i''^iPtfi{x). Hence Pt is a cone map on 71+. 
Extend by linearity the definition of Pt to TZ = TZ+ — TZ+. Denote the extension again by 
Pf It is easy to see that this extension is well defined and that / — )■ Ptf{x) is a linear 
functional on TZ. Moreover, by linearity, t — )■ Ptf{x) is right continuous and 



X) 



POO 

/ e-^'Ptfix)dt, fen. 

Jo 



Using the first property in ( 5.2), we see that 
d 



dp 



13'' 



/3 



— /-n!(-l)"(f/^+")"+V 
^^n\ [f - i/SU^^Tf] ■ 



For / e n+, choose a so that / is a-supermedian. Then we can iterate the inequality 
pjj/B+aj' < J to conclude that | — U^~^°'f is completely monotone. Since it is clearly the 
Laplace transform of / — e~°'^Ptf, we see that e'^'^Ptf < f for all t > 0. 

Since n contains the constant functions and [/"I = we see that P^l = 1 for all 
t > 0. 



CONSTRUCTING STRONG MARKOV PROCESSES 



13 



If / e 7^ and / is nonnegative, then (^-^j U^f = and since (U^Y+^f > 

we see that (3 — )■ If^f is completely monotone. Since U'^f is the Laplace transform of 
Pff, we see that Ptf > for all t > 0. Hence Pt is a positive linear operator on TZ and, 
since P^l = 1, it is bounded and therefore continuous. We can now extend the definition 
of Pt by continuity to cl{TZ). Fix t and x and consider the hnear functional / — > Ptf{x) 
defined on Cq{E). Let be a sequence of nonnegative functions in Cq{E) that decreases 
to zero pointwise. Then by Dini's lemma /„ converges uniformly to zero. Hence Ptfn{x) 
converges to zero and so we can apply Daniel's theorem to conclude that there exists a 
unique measure Pt{x, •) on B such that Pt{x) — J Pt{x, dy)f{y) for all / e Cq{E). 

By Proposition A. 4, the map (t,a;) — )■ Ptf{x) is S([0, oo)) x B measurable for each 
f E TZ. Applying the lattice version of the monotone class theorem, it follows that 
(t, x) Ptf{x) is iB([0, oo)) X B measurable for every f & B and also that 



^0 



e-'''Ptf{x)dt 



for every f E B. 

Finally, we must show that Pt forms a semigroup. Again by a monotone class argument, 
it suffices to show that PtPsf = Pt+sf for all / G 7^+, and s.t > 0. Fix / G TZ+. Then 
PtPgf and Pt+sf are both right continuous in s and so it suffices to show that, for each 
t > 0, they have the same Laplace transform on s. That is. 



PtW^fix) 



e-'''Pt+sf{x)ds. 



Since C/°/ G TZ, the left-hand side is right continuous in t. So is the right-hand side. 
Hence, it suffices to check that they have the same Laplace transform on t. Thus 



e-''' Pt+sf {x)dsdt 



POO P 

= / 

Jo Jo 

poo poo 

= / / e-^'^-"^'e-'''Psfix)dsdt 
Jo Jo 

Jo Uo 



Since 



we must check that 



- [1- 

—a I 



e-^^-^^'dt 



Psf{x)ds. 
(3 = a 



{U^ff ^ = a 
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which is true since [/" is a resolvent. This completes the proof. □ 



The transition function constructed in Theorem 3.7 will be called an Engelbert tran- 
sition function. Let W denote the space of all functions from M"*" = [0, oo) into E that 
are right continuous and have left-hand limits. Put Yt{w) = w{t) for w G W and let 
go = a{Yt\t > 0} and = a{Yt\0 <s<t}. 

3.10 Theorem. Let pt be an Engelbert transition function. Then, for each probability 
measure fi on {E, B), there exists a probability measure on (W, Q^) such that 

(i) E^{f{Yt+sWJ = Ptf{Y,), a.s. for / G S and all s,t>0; 

(ii) EJ{Yt) = J fi{dx)Ptf{x), for feB and alH > 

(E^Z denotes the expectation of Z{w) with respect to the probability measure P^). 

Proof. The proof is quite standard and we will present essentially the same outline as in 
[n], Theorem 5.1. Let Q = ^1°'°^) and J" = B^°''^\ Put Xt{u) = uj{t) and J^t = (t{X,|0 < 
s < t}. Given /i, we can apply the Kolmogorov extension theorem to obtain a measure P 
on (fi, J-") such that (Xj, J^t,P) satisfies properties (i) and (ii). 

We must show that this process has a modification that is right continuous and has 



left-hand limits. By Theorem 3.7-(ii), for each / G TZ+, there exists an a > such that 
Q-atp^j- ^ j- 'pj-^is fact together with (i) implies that e^"*/(Xt) is an J-'^-supermartingale. 
Hence, with P-probability one, the map t — )■ f{Xt) restricted to the positive rationals 
Q"*" has left- and right-hand limits at each point in M"*". Since cl(7^) D Co(-E), it follows 
that for each / G Co{E), with P-probability one, the map t — )■ f{xt) restricted to has 
left and right-hand limits at each point in M"*". The space Cq{E) contains a countable 
subset (/n)n>o that separates the points in E. Let iV„ be a P-null set off of which fni^t) 
restricted to has left- and right-hand limits at each point of M"^. Put N = U Nn. Since 

n 

(/n)n>o separates points, it follows that, on Q\N, the map t — )• Xt, restricted to has 
right- and left-hand limits in E at each point in Take Q\N as our new sample space 
Q. For each t G and u E Q, put 



Xt+{oj) = lim Xr{ijj). 

r-lt 
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Fix f,gE 7^+, t G and let Vn be a sequence in Q+ such that r„ 1 1. Then, applying 
(ii) we get 

Ef{Xt)g{Xt+) = lim E/(Xi)(7(X,J 

n~^oo 

= \imEf{X,)Pr^.tg{Xt) 

n^oo 

3.11 = Ef{Xt)g{Xt) 

where g{x) = limt^Q Ptg{x) = Pog{x). Since e'^'^Ptg < g (for some a > 0) and g > 0, it 
follows that < g < g. Since Ptg = PtPog = Ptg, we see that 



Eg{Xt) = J ii{dx)Ptg{x) = J ii{dx)Ptg{x) = Eg{Xt). 
Hence g{Xt) = g{Xt) P-a.s. and so ( 3.11[ ) implies that 



3.12 Ef{Xt)g{Xt+) = Ef{Xt)g{Xt 



Since cliJZ) D Co{E), ( 3.12) actually holds for all f,g E Co{E). A monotone class 



argument now shows that EF{Xt, Xt+) = EF{Xt, Xt) for allF eBxB. Taking F{x, y) = 
lx=y, we see that 

F{Xt = = 1. 

Now define Yt{uj) = Xt-^-{u}) for each t G M^, u E Q. Then the map t — )■ Yt{u) is right 
continuous on IR+ and has left limits on (0,oo). Put Ht = cr{Yt\s < t). Since Yt = Xt 
P-a.s., it follows that (l^,'Ht,P) satisfies (i) and (ii). Finally, we define a map vr from 
Q into W by the formula iT{uj){t) = Yt{u). Then the measure P^ = P o tt"-*^ has all the 
properties stated in the theorem. □ 



Formula ( 3.10 )-(i) is the basic form of the Markov property. We now extend this 



property in the usual way. Indeed, for each t > let 6t denote the shift operator in W: 

{9tw){s) =wit + s). 

For each probability fi on {E, B) let denote the completion of with respect to P^ 
and let N^' = {B e G^" \ fJB) = 0}. Put = a{g^UN^}, Gt = n^f and ^ = U^^. 
For any bounded ^-measurable function Z, put 



6tZ{w) = Z{9tw). 
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Using a standard monotone class argument, formula ( 3.10 )-(i) can be extended as follows: 



3.13 



Formula ( 3.13) is called the Markov property. 

A map r from W to [0, oo] is called a stopping time if {w \ t{w) < t} ^ Qt for all t > 0. 
If r is a stopping time, we define the pre-r a-algebra Qr to be the collection of all sets 
A E Q such that y4n{r < t} E Qt for all t >0. The following theorem extends ( 3.13) to 
stopping times. 



3.14 Theorem. Let be the extension to (W, Q) of the measure constructed in Theorem 



3.10 Then, for each stopping time r and for each Z G Q, 



3.15 E^,{9^Z\g^} = Ey^Z a.s. on {r < oo}. 

Also gt= nGs. 

s>t 

Proof. The proof follows immediately from the results proved in [Ij Sections 1.6-1.8. We 
only remark that in Theorem 1.8.11 of [1] we must replace his L by our class TZ. Then in 
his proof, the monotone class theorem can still be used since (j(7l) = B. □ 



Formula ( 3.15) is the strong Markov property. 



4. Examples 

4.1. Uniform Motion. Let {E,B) = {M.,B{M.)) and let pt be the transition function for 
uniform motion: 

Pt{x, B) = 1b{x - t). 

Then 

Ptf{x) = f{x-t) 

and 



It is clear that, for every f E B, U°'f G C(]R). Since C(]R) is a linear lattice, it follows 
that 7^ C C(M). 

Now suppose that g G C^(M) and put f = g' + ag. Then, a simple calculation shows 
that t/"/ = g. Hence, C^{M) C C{M). The classes C^{M) and C(M) generate 
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the same uniformity and this uniformity is also generated by = {e(x) = x}. Hence the 
intrinsic state space coincides with the real line. 

4.2. Brownian Motion. Let {E,B) = (R.''', B(W^)) and let pt be the transition function 
for (i-dimensional Brownian motion: 



Pt{x, B) = j Ptiy - x)lB{y)dy 

where 

Pi(y) = (27rt)-'^/2exp{-|y|V2t}. 

Then 

It is easy to check that for every e > there exists a < oo such that: 

\ptiy-x) -ptiy-xo)\ < Kp2tiy-xo), 

whenever |x — xq] < e. Hence, the dominated convergence theorem shows that for every 
f eB, Ptf belongs to C{R'^). Put g = f/"/. By Proposition | 2.10f (i), lim^o ||^t^-^|| = 0. 
Since x — >■ Ptg{x) is continuous, it follows that g belongs to C(]R'^). Thus, C C C(]R'^) and 
since C(M'^) is a linear lattice we have C dTZd CiW'-). 

Let (yf be a function of class C'^{W^) and let / = — ^A^f + ag where A = ^ + ■ ■ ■ + ^ 
is the Laplacian in W^. If we integrate by parts (in x), then use the fact that \^Pt = ^Pt, 
and integrate by parts again (in t), we get 

e-"'PJ{x)dt = e-"^ j ps{y - x)g{y)dy. 

Letting s J, and using the continuity of g we get ?7"/ = g. Hence, C'^iW^) d C d IZ d 
C{W^). Since the classes C'^iW^) and CiW^) generate the same uniformity on we see 
that we may take 

£ = {ei{x) = xi, e2(x) = Xa, . . . , ed{x) = Xd}. 
Hence the intrinsic state space coincides with R'^. 

4.3. Pure Jump Process. Let {E,B) be a general state space, let g be a stochastic 
kernel on {E, B) and put 



n=0 



p,{x,B) = J2e'''^-^q^-\x,B). 
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Then 



and 



n=0 



ni 



a-\- X 



E 

n=0 



A 



a + A 



where Q is the operator on B corresponding to q. 

Suppose that g E B and put / = A(l — Q)g + ag. Then a simple calculation shows 
that U"f = g. Hence B G C and so it follows that B = C = TZ. The class B generates a 
finite dimensional uniformity if and only if £' is a countable set. In this case, if points are 
measurable, then we can take E = {e{xn) = n} which maps E into the set of nonnegative 
integers. 

4.4. Uniform Motion with Sticky Origin. Let {E,B) = (R,S(R)) and let pt be the 

transition function for uniform motion with "stickum" at zero (discussed in the Introduc- 
tion): 



Pt{x,B) 



pT—X 

'1b(0)+ / e~nB{x - 1 + u)du 0<x<t 
Jo 

Isix — t) else. 



Then 



Uy{x) 



1 + a 



/(0)+ / e"7(t)cit + (1 + a) / e"'f{t)dt 



e'''f{t)dt 



x>0 



X <0. 



It is easy to see from this formula that ii f E B then C/°/ belongs to C((— oo, 0), [0, oo)). 
Since this last class is a linear lattice, it follows that TZ C C((— oo, 0), [0, oo)). 

Now suppose that g G C^((— oo,0), [0, oo)) and g'{0) = (this is the right-hand deriv- 
ative at zero). Put f — g' + cyg- Then a simple integration by parts shows that t/"/ = g. 
Hence, 

C^((-oo,0),[0,oo))n{/i I /i'(0) = 0} C C C n C C((-oo,0),[0,oo)) 

where by h'{0) we mean the right-hand derivative at zero. It is easy to see that we can 
take S consisting of one function 



e{x) 



X 

X — 1 



X > 0, 
X <Q. 
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This function makes small negative numbers "far away" from small positive numbers. 
Also the space — (— oo, — 1) U[0, oo) is not closed and so the intrinsic state space 

E' — (—00, —1] U[0, 00) turns out to have one point more than E. 

4.5. Uniform Motion with a Jump. Let E = (-00, -1] U[0, 00) = E~ U E'^ and let 
B be the Borel cr-algebra on E. Let pt, be "severed" uniform motion: 



Pt{x, B) = 



Then 



lB{x-t-l) 

Ib{x - t) 



f{x-t-l) 
fix - 1) 



0<x<t, 
else. 



0<x <t, 
else, 



and 



X e E+, 
X e E~. 



Note that C/°/(0) = C/"/(-l). Put 7_i,o = {h \ h{-l) = /i(0)}. We see that if / e B 
then belongs to C{E) n /_i,o. 

Now suppose that g e C^{E) n/_i,o and put f — g' + eng. Again, integrating by parts 
shows immediately that g — U°-f. Hence 

c^(E)n/_i,o c c c 7^ c c(E)n/_i,o. 

In this example, TZ does not separate —1 and 0. We may choose for E the collection 
consisting of the single function 



e{x) 



X 

x + 1 



X eE+ 
X e E~. 



On the intrinsic state space the process is just uniform motion. 

4.6. Uniform Motion with Fork in the Road. In R^, let R denote the closed right 
half-line on the a;-axis, let U denote the open upper half-line on the y-axis, and let L 
denote the open lower half-line on the y-axis. Let E = RUVUL and let B be the 
Borel cr-algebra on E. Let pt be the transition function for the process that proceeds 



20 



ROBERT J. VANDERBEI 



deterministically left, up, or down on R, U, or L, respectively. When it arrives at the 
origin, it proceeds upward/downward each with probability 1/2. Then 



f{o,y-t) 

f{x-t,0) 
[ 1/(0, + 1/(0, x-t) 



and 



-at 



f{0,t)dt 



e"'f{0,t)dt 



e'^'fit, 0)dt 



{x,y) e U, 
{x,y) e L, 

{x,y) e R, <t < X, 
{x,y) e R, <x < t, 

{x, y) e U, 
{x,y) e L, 



e"M/(0,t) + /(0,-t)]rft {x,y)eR. 



From this formula for f/", wc see that C G TZ G C{R, U,L). 

Now suppose that g e C\R, U, L) and ^(0, 0) = | [^(0, 0+) + ^(0, 0-)]. Put 

dg 



f 



dx2 
dg 

0X2 

dg 
dxi 



+ oig 
+ ag 
+ ag 



(x, y)eU 
(x, y)eL 
{x,y) e R. 



Then, as in the previous examples, simple calculations show that g = f/"/. Hence g G C. 
Since the collection of all such functions g generates the same uniformity as C{R,U,L) 
does, we see that we can take E = {ei{x , y) , e2{x , y)} where 

ei{x,y) = lR{x,y) 

( y + 1 {x,y)GU 
e2{x,y) = < 1/ - 1 {x,y) gL 
[ {x,y)GR. 

Hence ipi^E) = {{x,y) \ x > 0,y = or x = 0,y > 1 or x = 0,y < —1} and so the closure 
— '4>{E) contains two new points: (0, 1) and (0, —1). Since the intrinsic strong Markov 
process is right continuous we see that, with probability one, it visits one of these two 
new points and it never visits the branch point (0,0). 
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4.7. Brownian Motion with an Absorbing State. Let {E,B) = {R'^,B{R'^)) and let 
Pt be the transition function for rf-dimensional Brownian motion except it is ahered at 
X = as follows: 

J Pt{y - x)lBiy)dy x^O 
1b(0) X = 



Pt{x,B) 



where pt{y) is as defined in Subsection 4.2[ Since the origin is a set of measure zero, it is 



easy to check that this pt is a Markov transition function. It is also easy to see that TZ 
coincides with the TZ for Brownian motion except that the value of each function at zero 
can be changed arbitrarily. Hence we may take 

S = {ei{x) = xi, e2{x) = X2, ed{x) = Xd, ed+i{x) = lo(x)}. 

Hence the intrinsic state space is a closed subset of M"*"*"^ that consists of two parts: the 
(i-dimensional hyperplane corresponding to Xd+i = and the single point corresponding 
to Xd+i = 1 with the rest of the coordinates vanishing. On the hyperplane the transition 
function is that for ordinary Brownian motion. The single point is an absorbing point. 

4.8. Instantaneous Jump to Limiting Distribution. Let {E,B) be a general mea- 
surable space and let pt{x, B) = n{B) where vr is a probability measure on {E, B). Then 



Ptfix) = J f{y)7T{dy) 

and 

UV{x) = - [ f{y)7T{dy). 
a J 

Hence, for f E B, U°'f is a constant function. Hence, both C and TZ coincide with the 
collection of all constant functions. In this example, every point gets identified as the 
same. We may take 8 consisting of one function 

e(x) = 1. 

In this case, the entire state space collapses to a single point. 

5. Appendix 

In this section, we collect certain results that are either well-known or secondary to the 
main theme of the paper. 



5.1 Proposition. Let E he a, closed subset of and let be a class of continuous 
real-valued functions defined on E. Then the a-algebra generated by 1-L coincides with 
the Borel a-algebra on E if and only if "H separates points in E. 
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Proof. The "only if" direction is trivial. Suppose that T-L separates points in E. For 
a G and r > 0, let Bj.{a) denote the intersection of E and the closed ball of radius r 
centered at a. Fix a & E and < vq < ri. Put An = {x ^ E \ nri <\x — a\ < {n + ^)ti}. 
Fix X G B.i.q{o) and y G Br^^{aY. Let / be a function in 1-i for which f{x) ^ fiy)- 
Suppose that /(x) < fiy). Pick s such that /(x) < s < /(y) and let V^^y = /^^{{—oo, s)) 
and Wx,y = /^^((— oo, s]). Then, in the relative topology, V^^y is open, Wx,y is closed, 
X G Vx^y C VTx.y and y G M/^y. The collection {V^^y \ x G i?ro(o)} forms an open 
cover of the compact set Br^ and hence there is a finite subcover K.^^^,. . . ,Vx„^y. Put 

n 

Fy = U Wxi^y Then Fy is a closed set, y E F^ and -Bro(«) C F„. Also, Fy G a{T-L). 

Now fix > 1. The collection {F^ \ y G A^} forms an open cover of the compact set 
An- Hence there is a finite subcover {Fy^^ | j = 1, 2, . . . , fc„}. The collection {Fy^ ^^ \ j = 
1, 2, . . . , fc„; = 1, 2, . . .} is a countable collection that covers {x | |x — a| > ri}. Hence, 

B,,Xa) C nF^^,„ C Br,{a) 

J,n. 

and n Fy^^ is evidently (T('H)-measurable and closed. Since tq and ri were arbitrary. 



we see that for any < tq < ri we can find a closed cr('H)-measurable set G such that 
Bro{a) C G C Br^{a). 

Now fix r > and put B = B^{a) and Bn = -Br+i/n(ct)- There exist closed o"('H) 
measurable sets Gn such that B C G„ C i?„,. Hence i? C r\Gn C n-B„ = B and so 

n n 

B = n Gn is a (T('H)-measurable set. Since a and r were arbitrary, we see that {Br{a) \ a G 

n 

E, r > 0} C (t{T-L) and since these sets generate the Borel a-algebra it follows that 
B[E) C a{T-L). Since every function in T-L is continuous, we see that cr('H) C B{E). □ 



The following properties follow directly from the resolvent identity ( 2.5). 



5.2 Proposition. Let f/" be a resolvent on {E,B). Then for each f e B and x E E, 
the map a -^^ U^flx) is infinitely differentiable on (0, oo) and 



f/"/ = (-l)"n!(f/")"+7 



and 

da 



{aU"f) = (-l)"+in!(t/")" [1 - at/"] /. 



Proof. The proof can be found in Getoor ^Jj page 7. The next two propositions are also 
proved in [TT] on pages 5 and 12, respectively. □ 
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5.3 Proposition. If / is /3-supermedian, then the map a — > {a — P)U°'f{x) is increasing 
for q; > /? and the hmit / = hmQ,_^oo aU°'f is /3-supermedian. Also f < f and C/"/ = C/"/ 
for all a > 0. 
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